1. Field of the Invention
The present invention relates generally to image/video display and processing systems, and in particular, to a method, apparatus, and article of manufacture for conducting paint operations in a procedural renderer and revealing elements in a paint buffer.
2. Description of the Related Art
In a multi-layer context (e.g., a compositing application), a user may paint (e.g., a brush stroke) on any particular layer. However, if a paint operation contains multiple different elements, the prior art requires the creation of a result clip prior to being used in a composite. In addition, if the user is editing a clip in a procedural tree, the prior art does not permit the user to directly edit a paint operation. Instead, the prior art requires the user to exit the procedural tree, utilize a paint application, perform the edit, save any changes, and re-enter the procedural tree again. Such limitations are inefficient and inflexible. These problems may be better understood by describing prior art composites, paint operations, and reveal operations.
In a compositing application, multiple layers of footage are blended together into a single composite. For example, using a stage theater as an analogy, a user may start with a backdrop, and then add the scenery layer by layer, from back to front, until the scene is cohesive and complete. Accordingly, a layer is a 2D object that can be arranged and animated in 3D space.
To edit a layer or to create particular images on a layer, a paint application may be used. In audio/video applications, a paint application enables the user to perform a wide range of painting tasks from simple retouching, to motion tracking, to full cartoon animation. Accordingly, a paint application is an interactive vector-based and object-oriented drawing operator. In paint applications, a user may create projects for painting on changing video frames (and not on a still). Accordingly, the paint object must move and follow the action in a scene. In addition, a paint operator may be applied to a particular layer (or group of layers).
Within a paint application, an object consists of anything that may be drawn with a paint drawing operator. Objects may be divided into three main groups: paint, selection, and mask. Paint objects create or modify pixels. Selection objects define a specific area of the screen within which other paint objects or effects operate. Mask objects make areas of a layer transparent. Thus, after, an object is created, the object may be selected and its properties (e.g., size, shape, color, opacity, etc.) may be changed. Examples of objects that may be created in a paint application include: brush strokes, rectangles, ellipses, and polygons; text; lasso selections; magic wand selections; rectangular, elliptical, and polygon selections; text selections, fill; freehand masks; and rectangular, elliptical, and polygon masks.
The stacking order refers to the placement of the objects in a stack. Objects at the bottom of a stack are covered by objects at the top of the stack. Selections may act on objects above them in the stacking order until another selection is encountered. Thus, if a selection is created on top of an existing selection, the new selection becomes the active selection and paint operators only apply to the active selection.
A mask may be used to isolate a region of an image, and make the rest of the image completely transparent. The mask affects the image's alpha channel, which is where the transparency information is stored. A mask may be used to create an alpha channel, or to add to an existing alpha channel. Objects below a mask in the stacking order are transparent; objects above the mask in the stacking order may or may not be transparent depending on their own alpha information. For example, referring to FIG. 1, if an ellipse 102 is drawn (on top of a mask 104) without an alpha channel (e.g., only RGB), then the shape will not change the layer's transparency and therefore will not be visible outside the mask 104. Thus, the area of ellipse 102 outside of the mask 104 is completely transparent. If an ellipse is drawn with RGB and its own alpha channels (RGBA) 106, then the shape 106 “overwrites” the mask's 104 alpha information, and the ellipse 106 is visible outside of the mask 104. To isolate a region of an image without affecting the transparency, a selection should be utilized instead of a mask. When an operator (e.g., a color corrector) is applied above the selection, only the isolated area is affected and the rest of the image is still visible.
As described above, alpha channels are used to determine how much of a front image shows over a back image. For example, if the user is looking through a keyhole inside a room, the door is the front, the keyhole is the alpha, and the room is the back. Combined together, the layers form a composite. If the contents of the alpha channel are rendered, the result is referred to as a matte or matte layer. A matte can be completely black and white (called a “hi-con” for high-contrast), or can contain many shades of gray. In video and film, mattes are rarely static, and are rarely pure black and white. Any areas on the front containing transparency (e.g., glass, smoke, curly blonde hair, etc.) or motion blur will appear in the matte as gray areas, allowing the transparent (or blurred) areas to partially show through.
As described above, a paint application may be used in conjunction with a composite to paint on one layer or over several layers at a time. For instance, a paint application may be used to change a subject's eye color, remove wires, fix video drop-out on a selected layer, or create text effects and cartoon animations that are added over all of the layers in a composite. In addition, prior art paint applications include both vector paint application (that define strokes as objects) and raster paint applications (that modify pixels in the image directly or write pixels directly in separate layers that may be combined in a source image thereby modifying the source image).
Paint operators are like any other operator in that they may be viewed schematically on a per-layer basis. FIG. 2 illustrates a process tree in a schematic view. A schematic view 200 uses nodes 202-214 to represent all of the operators in a workspace performed on a particular layer, and lines (edges) 216 between nodes 202-214 to indicate the flow of image data between the operators. The output of one node 202-214 is the source for the next node 202-214. As composites 212 or other branches (e.g., paint or color correction projects) are created, a process tree indicating the flow of data through the nodes 202-214 is built automatically. As illustrated in FIG. 2, source data consists of gator node 202 and snake node 204. A paint operation (e.g., via paint node 206) is performed on gator node 202 (e.g., by drawing the sun). The connection nodes 208-210 are then used as input to the composite operator node 212 which performs a compositing operation. The compositing operation node 212 produces a result/output 214.
In addition, prior art paint modules in procedural renderers (i.e., conducted in a schematic view) are raster paint nodes that do not have layers and modify their input buffers directly. In this regard, an input buffer in an procedural renderer may come from a fairly complex process tree and modifications to that tree will re-create the input buffer. A side-effect of these modifications is that the paint strokes, being applied directly to the input buffer, are lost when that input gets modified through a change in the nodes upstream of a paint node.
As used herein, an operator (e.g., paint operator 206) is the most common node. A connection node 208 and 210 represents an input for an operator. In the case of a composite operator 212, the visible connection nodes 208 and 210 are layers. The output node 214 is a node representing the render queue settings for an operator. Lastly, several nodes may be grouped into a group node.
The process tree (such as the process tree illustrated in FIG. 2) can be used to manage operators and nodes. For example, users may add operators (e.g., a paint operation) to a branch, change the process order of operators 206, and navigate easily between operator controls. Further, if an operator is added to a layer in a schematic view 200, the operator will appear as a node (e.g., the paint node 206). The same operator node may also appear in other workspace environments (e.g., a workspace panel). Further, in the schematic view, particular operators may be turned on/off as desired simply by clicking on the icon for that operator (e.g., paint 206 or composition 212).
While the above prior art schematic view and paint operations may be used to modify and edit an image/clip, the prior art has many limitations. For example, in some prior art applications, a paint operator 206 is associated with a particular version of a source image 202. Thus, if a source image (e.g., image 202) is edited, the paint operator 206 disappears and all information associated with the paint node 206 is lost. In addition, prior art paint operations cannot be performed from within the schematic view 200. Thus, in order to perform any modifications using a paint operation, the prior art requires the user to exit the schematic view, enter the paint application, perform the desired paint operation, produce a result clip, reenter the schematic view 200, import the result clip into the schematic view 200 and continue.
Prior art applications may also provide for the use of a reveal tool. Revealing is similar to using an eraser on one layer, allowing the user to see what is beneath it. Typically, the reveal tool is used for adding or removing objects from a scene (also referred to as rotoscoping) (e.g., scratches, wires, or a microphone boom). Phrased in another manner, the reveal tool allows the user to use a paint operator on a current frame and reveal an image from a designated source frame/clip (i.e., the user is revealing, via the paint operation, to another layer/source). The prior art reveal tools may be used in two ways: to reveal from a frame in the current clip; or to reveal from a frame in another clip.
A user may reveal to the current frame as a fast mechanism for copying or removing objects in the current frame. For instance, a user can perform a quick wire removal of an object in front of a green screen by revealing to the current layer with a slight position offset. When the user paints over the wires (using a paint operator), the user sees through to the green screen. For a uniformly lit green screen, the user can no longer distinguish where the wires were. Thus, there are various settings for a reveal operation.
FIG. 3 illustrates a dialog window indicating the various settings for a reveal operation. Within the modes controls 302 on the paint controls panel 300, the user has selected Reveal 304. The Object property 306 allows the user to select a paint tool in the form of a particular shape (e.g., a rectangle, brush stroke, etc.) as well as whether the object is filled (e.g., a filled rectangle). The channels property 308 allows for the selection of an item in a list to draw on a single color channel. When the user switches channels, the viewport may update to display the specified color or alpha channel(s) for the selected object(s). Thus, if the blue channel is selected from the channel list 308, only shades of blue may be available on different panels, and only the blue portion of the selected object's color is displayed in the viewport. In FIG. 3, RGBA has been selected. With RGBA selected, all of the colors (red, green, blue) and the opacity (alpha) will be available on different panels and will be displayed in a viewport.
The source property 310 allows the user to pick the layer to use as the reveal source. If “current” is selected, the footage currently loaded into the paint application is used as the source. A pick operator may be used to access a dialog that allows the user to choose another operator (e.g., from the process tree of FIG. 2) as the reveal source. From the dialog, the user can import footage from disk for use as a reveal source or select an operator corresponding to the image or clip the user desires to use as the reveal source.
The frame property 312 allows the user to enter a frame number corresponding to the frame to use from within the source specified in source property 310. In addition, the user may elect to lock 314 the reveal source. If lock 314 is enabled, the reveal operation only reveals to the frame specified in the frame field. If the user desires the reveal source to advance and reverse frames as the user moves through the clip, the lock 314 should be disabled. Because the reveal operation reveals to the pixels that are directly underneath, the user may want to adjust the position of the reveal source with respect to the target layer. To position the reveal source, the user can click and drag the set position button (i.e., the “+”) to position the source layer. Alternatively, the user can enter X and Y offset values in the corresponding fields.
An example of the use of the offset field 316 is when the user desires to remove wires (e.g., used to hang a model airplane for a flight scene). To accomplish such removal, the user would specify the current layer as the reveal source 310 and offset the frame slightly so that when the user paints over the wires, the adjacent pixels are revealed on the target image.
Another example of the use of the reveal operation is if the user desires to delete an unwanted object that appears momentarily in a clip. Such an object may be removed by revealing through to another frame at a time that the object was not present. The user would load the layer into paint, advance to the first frame in which the object is present, click on reveal 304, enter a frame number corresponding to the frame in which the object is not present and ensure that the lock 314 is enabled. A paint tool may then be used to remove the unwanted object and the user advances to the last frame in which the object is present. Accordingly, the revealed area in the single frame is used on every frame in the clip having the unwanted object.
In yet another example, the user may desire to reveal from one clip to another (e.g., to remove an element that is always present in the scene (e.g., a moving plane). If the user has a second clip shot in front of the same background in which the unwanted object is not present, the second clip shot can be used as the reveal source. To accomplish such a reveal operation, the user imports two layers into a composite and specifies the topmost layer as a target layer and a bottom layer as the reveal source 310. After selecting the top layer, the reveal button 304 is selected and a source image other than the current layer is selected for the reveal operation. The frame number containing the image to reveal is entered in frame field 312 and the lock 314 is enabled to reveal only to the specified frame (lock 314 is disabled if the user desires the revealed source to advance and reverse frames as the clip is advanced). The user can then paint as desired to reveal to the specified source.
While the reveal operation has many uses, the prior art reveal operations have various limitations. For example, to reveal from a source to a target as described above, a source/target clip must first be created and then added to a composite. Accordingly, the user cannot select particular elements from different layers to reveal to. In addition, the user cannot select particular elements on the same layer in a composite to reveal to. Further, once the user selects a particular layer as a source, the user cannot provide further limitations regarding the alpha channel on that layer (e.g., whether the reveal option will maintain the alpha information from the target or source).
In view of the above, what is needed is the ability to conduct a reveal operation while presenting the user with more flexibility and options. Further, what is needed is the ability to perform a paint operation from within a schematic view.